Enhance SSN undocumented type imputation #265

vahid-ahmadi · 2025-05-23T16:34:06Z

Fixes #246

Implement ASEC Undocumented Algorithm (paper)

Algorithm Logic: Process of Elimination

Start: Assume all foreign-born non-citizens might be undocumented (code 0)
Remove: People with evidence of legal status → move to code 3 ("OTHER_NON_CITIZEN")
Result: Those remaining in code 0 have no clear indicators of legal status → likely undocumented

Target Implementation

Modify the existing add_ssn_card_type() function to apply these conditions before the random refinement step, ensuring that people meeting any of these conditions are assigned to code 3 ("OTHER_NON_CITIZEN") rather than potentially remaining as code 0 ("NONE"/undocumented).

The 14 Conditions

Condition 1: Pre-1982 Arrivals

Logic: Remove those who arrived before 1982 (eligible for IRCA amnesty)
Variable: PEINUSYR codes 1-7 (Before 1950 through 1980-1981)

Condition 2: Eligible Naturalized Citizens

Logic: Remove naturalized citizens who meet time/age requirements
Variables: PRCITSHP == 4, A_AGE >= 18, PEINUSYR (for years in US), A_MARITL, A_SPOUSE

Condition 3: Medicare Recipients

Logic: Remove those with Medicare coverage
Variable: MCARE == 1

Condition 4: Federal Retirement Benefits

Logic: Remove those receiving federal government pensions
Variables: PEN_SC1 == 3 OR PEN_SC2 == 3 (Federal government pension)

Condition 5: Social Security Disability

Logic: Remove those receiving SS disability benefits
Variables: RESNSS1 == 2 OR RESNSS2 == 2 (disabled adult or child)

Condition 6: Indian Health Service Coverage

Logic: Remove those with IHS coverage
Variable: IHSFLG == 1

Condition 7: Medicaid Recipients (State-specific adjustments needed)

Logic: Remove Medicaid recipients (with state policy exceptions)
Variables: CAID == 1, GESTFIPS (for state-specific rules)

Condition 8: CHAMPVA Recipients

Logic: Remove those with CHAMPVA coverage
Variable: CHAMPVA == 1

Condition 9: Military Health Insurance

Logic: Remove those with TRICARE/military coverage
Variable: MIL == 1

Condition 10: Government Employees

Logic: Remove government workers and military personnel
Variables: PEIO1COW codes 1-3 (federal/state/local gov) OR A_MJOCC == 11 (military)

Condition 11: Social Security Recipients

Logic: Remove those receiving Social Security payments
Variable: SS_YN == 1

Condition 12: Housing Assistance (State-specific adjustments needed)

Logic: Remove housing assistance recipients (with mixed-status family exceptions)
Variables: Household-level HPUBLIC == 1 OR HLORENT == 1, GESTFIPS

Condition 13: Veterans/Military Personnel

Logic: Remove veterans and active military
Variables: PEAFEVER == 1 OR A_MJOCC == 11

Condition 14: SSI Recipients

Logic: Remove those receiving SSI for themselves (not on behalf of children)
Variables: SSI_YN == 1, RESNSSI1/RESNSSI2 (to verify recipient)

Additional Steps

Family Correlation Adjustment

MaxGhenis · 2025-05-23T16:53:12Z

How many undocumented do we get at the end of it (in the CPS)?

vahid-ahmadi · 2025-05-26T14:39:59Z

How many undocumented do we get at the end of it (in the CPS)?

Undocumented ssn pop in CPS:

Family correlation (80%) and random refinement (30%): 3.0 million
Family correlation (80%) and random refinement (10%): 4.1 million
Family correlation (80%) and random refinement (80%): 0.8 million
Family correlation (50%) and random refinement (50%): 2.1 million
Family correlation (100%) and random refinement (0%): 4.6 million

Also, here is the pop in each category before and after applying the 14 refinement conditions (original CPS data):

MaxGhenis · 2025-05-27T02:24:29Z

Ok we need to adjust to get the 11 million Pew estimates (might be more today)

vahid-ahmadi · 2025-05-27T11:45:43Z

Ok we need to adjust to get the 11 million Pew estimates (might be more today)

I adjusted the code to get the 11 million Pew estimates and the 2 million JCT estimates for the reconciliation reform.

policyengine_us_data/datasets/cps/cps.py

policyengine_us_data/tests/test_datasets/test_enhanced_cps.py

policyengine_us_data/utils/loss.py

policyengine_us_data/datasets/cps/cps.py

MaxGhenis · 2025-06-01T20:05:07Z

Also depending on the target year let's target the total undocumented population per these projections:

Year	Millions of people	Source / basis
2022	11.0	Official DHS Office of Homeland Security Statistics estimate for 1 Jan 2022 (Office of Homeland Security Statistics)
2023	12.2	Center for Migration Studies ACS‐based residual estimate (published May 2025) ([CMS]2)
2024	13.0	Reuters synthesis of experts ahead of the 2025 change of administration (“~13-14 million”) – central value used here (Reuters)
2025	13.0	Same midpoint carried forward – CBP data show a 95 % drop in new border apprehensions after Feb 2025, implying near-zero net growth for the year ([U.S. Customs and Border Protection]4)
2026	13.1	Author’s projection (see explanation below)
2027	13.2	“”
2028	13.3	“”
2029	13.3	“”
2030	13.4	“”
2031	13.4	“”
2032	13.3	“”
2033	13.2	“”
2034	13.1	“”
2035	13.0	“”

vahid-ahmadi · 2025-06-02T11:58:38Z

Also depending on the target year let's target the total undocumented population per these projections:

Year Millions of people Source / basis
2022 11.0 Official DHS Office of Homeland Security Statistics estimate for 1 Jan 2022 (Office of Homeland Security Statistics)
2023 12.2 Center for Migration Studies ACS‐based residual estimate (published May 2025) ([CMS]2)
2024 13.0 Reuters synthesis of experts ahead of the 2025 change of administration (“~13-14 million”) – central value used here (Reuters)
2025 13.0 Same midpoint carried forward – CBP data show a 95 % drop in new border apprehensions after Feb 2025, implying near-zero net growth for the year ([U.S. Customs and Border Protection]4)
2026 13.1 Author’s projection (see explanation below)
2027 13.2 “”
2028 13.3 “”
2029 13.3 “”
2030 13.4 “”
2031 13.4 “”
2032 13.3 “”
2033 13.2 “”
2034 13.1 “”
2035 13.0 “”

I've replaced the fixed 11 million target with dynamic targets.

vahid-ahmadi · 2025-06-09T15:01:12Z

nikhilwoodruff

Since this is getting quite complex, could you add a page to the documentation with descriptions of the methodology, and some statistics of our results? e.g. splits of status counts.

vahid-ahmadi · 2025-06-17T10:39:36Z

Since this is getting quite complex, could you add a page to the documentation with descriptions of the methodology, and some statistics of our results? e.g. splits of status counts.

I added documentation. Once the implementation is reviewed, I will complete the doc with results.

Remove git conflict markers and keep both test functions. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

Add explicit UTF-8 encoding when reading/writing documentation files to prevent UnicodeDecodeError on Windows systems. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

MaxGhenis · 2025-06-19T16:32:15Z

Please make the family adjustment probabilistic. It should only move people into undocumented status enough to hit the overall target in the CPS, if we don't have enough without it. Millions of US citizen children live with an undocumented parent ("mixed-status families") and the current algorithm results in zero such cases AFAIUI.

Also for reference here's the output from CI:

Step 0 - Initial: Code 0 people: 319,547,413

Step 1 - Citizens: Moved 296,159,040 people to Code 1

ASEC Conditions - Current Code 0 people: 23,388,373
Condition 1 - Pre-1982 arrivals: 1,144,841 people qualify for Code 3
Condition 2 - Eligible naturalized citizens: 0 people qualify for Code 3
Condition 3 - Medicare recipients: 1,653,008 people qualify for Code 3
Condition 4 - Federal retirement benefits: 13,045 people qualify for Code 3
Condition 5 - Social Security disability: 143,057 people qualify for Code 3
Condition 6 - Indian Health Service coverage: 21,500 people qualify for Code 3
Condition 7 - Medicaid recipients: 4,398,182 people qualify for Code 3
Condition 8 - CHAMPVA recipients: 4,972 people qualify for Code 3
Condition 9 - Military health insurance: 114,583 people qualify for Code 3
Condition 10 - Government employees: 690,024 people qualify for Code 3
Condition 11 - Social Security recipients: 1,200,030 people qualify for Code 3
Condition 12 - Housing assistance: 872,241 people qualify for Code 3
Condition 13 - Veterans/Military personnel: 69,[129](https://github.com/PolicyEngine/policyengine-us-data/actions/runs/15756429621/job/44412829970#step:7:130) people qualify for Code 3
Condition 14 - SSI recipients: 187,168 people qualify for Code 3
After conditions - Code 0 people: 15,991,100
  - Undocumented workers before adjustment: 10,931,508 (target: 8,300,000)
  - Undocumented students before adjustment: 901,242 (target: 399,000)
Step 3 - EAD workers: Moved 2,660,438 people from Code 0 to Code 2
Step 4 - EAD students: Moved 514,767 people from Code 0 to Code 2
After EAD assignment - Code 0 people: 12,870,904
Step 5 - Family correlation: Changed 1,326,999 people from Code 3 to Code 0 in households with Code 0 members
After family correlation - Code 0 people: 14,197,903
Step 6 - Target refinement: Moved 1,164,053 people from Code 0 to Code 3
After target refinement - Code 0 people: 13,033,850

Final populations:
  Code 0 (NONE): 13,033,850
  Code 1 (CITIZEN): 296,159,040
  Code 2 (NON_CITIZEN_VALID_EAD): 3,120,197
  Code 3 (OTHER_NON_CITIZEN): 7,234,327
Total undocumented (Code 0): 13,033,850 (target: 13,000,000)
Population log saved to: /home/runner/work/policyengine-us-data/policyengine-us-data/policyengine_us_data/datasets/cps/../../../docs/asec_population_log.csv
Documentation updated with population numbers: /home/runner/work/policyengine-us-data/policyengine-us-data/policyengine_us_data/datasets/cps/../../../docs/asec_undocumented_algorithm.ipynb

vahid-ahmadi · 2025-06-20T13:29:19Z

Please make the family adjustment probabilistic. It should only move people into undocumented status enough to hit the overall target in the CPS, if we don't have enough without it. Millions of US citizen children live with an undocumented parent ("mixed-status families") and the current algorithm results in zero such cases AFAIUI.

Addressed. Now we have the family adjustment probabilistic.

nikhilwoodruff · 2025-06-23T13:28:50Z

Documentation looks nice!

MaxGhenis

Looks great! And I misspoke earlier: we're not adjusting citizenship so won't get the wrong mixed-status household composition. But still good to do the other parts probabilistically since mixed-SSN-category households may exist.

MaxGhenis · 2025-06-05T16:28:59Z

policyengine_us_data/datasets/cps/cps.py

+    cps: h5py.File,
+    person: pd.DataFrame,
+    spm_unit: pd.DataFrame,
+    undocumented_target: float = 11e6,


Plus a flat 13/11 multiplier for next two

Suggested change

undocumented_target: float = 11e6,

undocumented_target: float = 13e6,

MaxGhenis · 2025-06-05T16:30:22Z

policyengine_us_data/datasets/cps/cps.py

+
+    if target_weighted_ead_workers > 0 and len(worker_ids) > 0:
+        # Sort workers by weight (heaviest first) to minimize assignments needed
+        worker_weights = person_weights[worker_ids]


MaxGhenis · 2025-06-05T16:40:17Z

policyengine_us_data/datasets/cps/cps.py

+        total_weighted_workers - undocumented_workers_target
+    )
+
+    if target_weighted_ead_workers > 0 and len(worker_ids) > 0:


create a function to modularize the targeting here: for each of {workers, students, all others}, compare the total potential undocumented to the undocumented target, then assign enough to documented to align them

add conditions

4c77463

vahid-ahmadi changed the title ~~add conditions~~ Enhance SSN undocumented type imputation May 23, 2025

add changelog

e143d61

vahid-ahmadi self-assigned this May 23, 2025

Merge branch 'main' into ssn-impute-new

e947c58

vahid-ahmadi added 6 commits May 27, 2025 10:41

add new calibration

0441bad

format

641e67e

edit tests

ec48a43

edit shape

a7804ad

edit condition

daa254c

delete prints

d5fe162

MaxGhenis requested changes May 27, 2025

View reviewed changes

vahid-ahmadi added 5 commits May 28, 2025 11:50

edit worker student condition

ae424f5

edit conditions

78ae416

edit condition 12

2a32696

edit cond12

8fdaed3

move code 2 part

e2de6f8

vahid-ahmadi requested review from MaxGhenis and removed request for MaxGhenis May 28, 2025 15:21

MaxGhenis requested changes Jun 1, 2025

View reviewed changes

MaxGhenis reviewed Jun 1, 2025

View reviewed changes

policyengine_us_data/datasets/cps/cps.py Outdated Show resolved Hide resolved

vahid-ahmadi added 3 commits June 2, 2025 11:00

put targets in cps.py

868d84c

edit workers and students logic

eefcf07

remove CTC Reform Impact Calibration

82123a4

vahid-ahmadi added 2 commits June 2, 2025 12:24

format

7ca653a

add years targets

79b06d7

change tolerance error

5cfff0e

vahid-ahmadi requested a review from MaxGhenis June 2, 2025 13:52

vahid-ahmadi added 2 commits June 9, 2025 15:41

edit logics and print pop

2eaa0c6

edit functions

365d0fc

nikhilwoodruff reviewed Jun 16, 2025

View reviewed changes

add documentation

fb78b59

vahid-ahmadi and others added 4 commits June 18, 2025 14:46

add documentation

a6dced5

Merge branch 'main' into ssn-impute-new

c38cef6

Fix merge conflict in test_enhanced_cps.py

e64df1a

Remove git conflict markers and keep both test functions. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

Fix Unicode encoding error in documentation update

c6253c4

Add explicit UTF-8 encoding when reading/writing documentation files to prevent UnicodeDecodeError on Windows systems. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

vahid-ahmadi requested review from MaxGhenis, PavelMakarchuk and nikhilwoodruff and removed request for MaxGhenis June 19, 2025 11:22

vahid-ahmadi added 3 commits June 20, 2025 12:04

add ctc reform into doc

ce739be

family adjustment probabilistic

b090af7

update documentation

2131da5

vahid-ahmadi added 2 commits June 23, 2025 12:27

edit doc

79205c3

package versions

b2f97f5

MaxGhenis approved these changes Jun 23, 2025

View reviewed changes

MaxGhenis merged commit d8afcf2 into main Jun 23, 2025
8 checks passed

MaxGhenis deleted the ssn-impute-new branch June 23, 2025 14:48

	undocumented_target: float = 11e6,
	undocumented_target: float = 13e6,

Enhance SSN undocumented type imputation #265

Enhance SSN undocumented type imputation #265

Uh oh!

Conversation

vahid-ahmadi commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Implement ASEC Undocumented Algorithm (paper)

Algorithm Logic: Process of Elimination

Target Implementation

The 14 Conditions

Condition 1: Pre-1982 Arrivals

Condition 2: Eligible Naturalized Citizens

Condition 3: Medicare Recipients

Condition 4: Federal Retirement Benefits

Condition 5: Social Security Disability

Condition 6: Indian Health Service Coverage

Condition 7: Medicaid Recipients (State-specific adjustments needed)

Condition 8: CHAMPVA Recipients

Condition 9: Military Health Insurance

Condition 10: Government Employees

Condition 11: Social Security Recipients

Condition 12: Housing Assistance (State-specific adjustments needed)

Condition 13: Veterans/Military Personnel

Condition 14: SSI Recipients

Additional Steps

Family Correlation Adjustment

Uh oh!

MaxGhenis commented May 23, 2025

Uh oh!

vahid-ahmadi commented May 26, 2025

Uh oh!

MaxGhenis commented May 27, 2025

Uh oh!

vahid-ahmadi commented May 27, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MaxGhenis commented Jun 1, 2025

Uh oh!

vahid-ahmadi commented Jun 2, 2025

Uh oh!

vahid-ahmadi commented Jun 9, 2025

Uh oh!

nikhilwoodruff left a comment

Choose a reason for hiding this comment

Uh oh!

vahid-ahmadi commented Jun 17, 2025

Uh oh!

MaxGhenis commented Jun 19, 2025

Uh oh!

vahid-ahmadi commented Jun 20, 2025

Uh oh!

nikhilwoodruff commented Jun 23, 2025

Uh oh!

MaxGhenis left a comment

Choose a reason for hiding this comment

Uh oh!

MaxGhenis Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

MaxGhenis Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

MaxGhenis Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

vahid-ahmadi commented May 23, 2025 •

edited

Loading